Optimized deferred lighting in a foveated rendering system

ABSTRACT

A method for implementing a graphics pipeline. The method includes determining a plurality of light sources affecting a virtual scene. Geometries of objects of an image of the scene is projected onto a plurality of pixels of a display from a first point-of-view. The pixels are partitioned into a plurality of tiles. A foveal region of highest resolution is defined for the image as displayed, wherein a first subset of pixels is assigned to the foveal region, and wherein a second subset of pixels is assigned to a peripheral region that is outside of the foveal region. A first set of light sources is determined from the plurality of light sources that affect one or more objects displayed in a first tile that is in the peripheral region. At least two light sources from the first set is clustered into a first aggregated light source affecting the first tile when rendering the image in pixels of the first tile.

CLAIM OF PRIORITY

The present application claims priority to and the benefit of thecommonly owned, provisional patent application, U.S. Ser. No.62/517,834, filed on Jun. 9, 2017, entitled “OPTIMIZED DEFERRED LIGHTINGIN A FOVEATED RENDERING SYSTEM,” which is herein incorporated byreference in its entirety.

TECHNICAL FIELD

The present disclosure is related to video games or gaming applications.Among other things, this disclosure describes methods and systems foraggregating light sources in a foveated rendering system.

BACKGROUND OF THE DISCLOSURE

Video gaming has increasingly become more popular with the advancementof video game technology. For example, high powered graphics processorsprovide an unbelievably viewing and interactive experience when playinga video game. In addition, displays are being designed with higher andhigher resolutions. For example, present technology includes displayshaving 2K resolution (e.g., 2.2 megapixels over 2048×1080 pixels) withan aspect ratio of approximately 19:10. Other displays having 4K UHD(Ultra high definition) resolution (e.g., 8.2 megapixels over 3840×2160pixels) with an aspect ratio of 16:9 are now pushing into the market andis expected to gain traction. Increased graphics processing capabilitiesalong with high resolution displays provide for a heretoforeunbelievable viewing experience for the user, especially when playing avideo game and gaming engine designed to take advantage of the higherresolution displays.

Pushing rendered images/frames to a high resolution display alsorequires increased bandwidth capabilities, such as between the renderingengine and the display. In most cases, a wired connection should be ableto handle the required bandwidth supporting the display. However, gamingsystems increasingly are configured with a wireless connection that mayprovide a bottleneck when pushing data to the display. For instance, awireless connection may be established between a gaming console local tothe user and the display. In these cases, the wireless connection maynot be robust enough to handle the required bandwidth to fully takeadvantage of the higher resolution displays, such that the video game asdisplayed may be interrupted (as the buffer is filling up) in order todisplay the entire video sequence as rendered. In some cases, the videogame processing may be throttled in order to match the lower bandwidthof the wireless connection to the display, such that the video framesmay be rendered at lower resolutions in order to push video data overthe wireless connection without interruption; however, by throttling theprocessing, the user is denied the full gaming experience with higherresolution graphics.

It would be beneficial to modify the graphics processing in order forthe user to achieve a high level of satisfaction for the user,especially when playing a video game.

It is in this context that embodiments of the disclosure arise.

SUMMARY

Embodiments of the present disclosure relate to foveated renderingconfigured to display portions of images in a foveal region with highresolution and portions outside the foveal region with lower resolution.In particular, image portions that are outside the foveal region arerendered with aggregated light sources, wherein an aggregated lightsource closely approximates the total effect produced from eachindividual light source of a corresponding cluster of light sources. Assuch, instead of computing the individual effects of each of theindividual light sources of a cluster, only the effect of the aggregatedlight source of the cluster is computed. In that manner, the totalbandwidth for the sequence of video frames being displayed is reduced,for example due in part to less complex images. Further, the sequence ofvideo frames can be delivered (e.g., over wired or wireless connections)in real time with minimal or no latency because the computationcomplexity is reduced.

In one embodiment, a method for implementing a graphics pipeline isdisclosed. The method includes determining a plurality of light sourcesaffecting a virtual scene. The method includes projecting geometries ofobjects of an image of the scene onto a plurality of pixels of a displayfrom a first point-of-view. The method includes partitioning a pluralityof pixels of the display into a plurality of tiles. The method includesdefining a foveal region of highest resolution for the image asdisplayed, wherein a first subset of pixels is assigned to the fovealregion, and wherein a second subset of pixels is assigned to aperipheral region that is outside of the foveal region. The methodincludes determining a first set of light sources from the plurality oflight sources that affect one or more objects displayed in a first tilethat is in the peripheral region. The method includes clustering atleast two light sources from the first set into a first aggregated lightsource affecting the first tile when rendering the image in pixels ofthe first tile.

In still another embodiment, a computer system is disclosed. Thecomputer system including a processor and memory, wherein the memory iscoupled to the processor and having stored therein instructions that, ifexecuted by the computer system, cause the computer system to execute amethod for implementing a graphics pipeline. The method includesdetermining a plurality of light sources affecting a virtual scene. Themethod includes projecting geometries of objects of an image of thescene onto a plurality of pixels of a display from a firstpoint-of-view. The method includes partitioning a plurality of pixels ofthe display into a plurality of tiles. The method includes defining afoveal region of highest resolution for the image as displayed, whereina first subset of pixels is assigned to the foveal region, and wherein asecond subset of pixels is assigned to a peripheral region that isoutside of the foveal region. The method includes determining a firstset of light sources from the plurality of light sources that affect oneor more objects displayed in a first tile that is in the peripheralregion. The method includes clustering at least two light sources fromthe first set into a first aggregated light source affecting the firsttile when rendering the image in pixels of the first tile.

In another embodiment, a non-transitory computer-readable medium storinga computer program for implementing a graphics pipeline is disclosed.The computer-readable medium includes program instructions fordetermining a plurality of light sources affecting a virtual scene. Thecomputer-readable medium further includes program instructions forprojecting geometries of objects of an image of the scene onto aplurality of pixels of a display from a first point-of-view. Thecomputer-readable medium further includes program instructions forpartitioning a plurality of pixels of the display into a plurality oftiles. The computer-readable medium further includes programinstructions for defining a foveal region of highest resolution for theimage as displayed, wherein a first subset of pixels is assigned to thefoveal region, and wherein a second subset of pixels is assigned to aperipheral region that is outside of the foveal region. Thecomputer-readable medium further includes program instructions fordetermining a first set of light sources from the plurality of lightsources that affect one or more objects displayed in a first tile thatis in the peripheral region. The computer-readable medium furtherincludes program instructions for clustering at least two light sourcesfrom the first set into a first aggregated light source affecting thefirst tile when rendering the image in pixels of the first tile.

Other aspects of the disclosure will become apparent from the followingdetailed description, taken in conjunction with the accompanyingdrawings, illustrating by way of example the principles of thedisclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure may best be understood by reference to the followingdescription taken in conjunction with the accompanying drawings inwhich:

FIG. 1A illustrates a system configured for providing an interactiveexperience with VR content, in accordance with one embodiment of thepresent disclosure.

FIG. 1B conceptually illustrates the function of a HMD in conjunctionwith an executing video game, in accordance with an embodiment of theinvention.

FIG. 2A illustrates a system providing gaming control to one or moreusers playing one or more gaming applications that are executing locallyto the corresponding user, in accordance with one embodiment of thepresent disclosure.

FIG. 2B illustrates a system providing gaming control to one or moreusers playing a gaming application as executed over a cloud gamenetwork, in accordance with one embodiment of the present disclosure.

FIG. 3A illustrates an image shown on a display and including a fovealregion of high resolution, wherein the foveal region corresponds to acenter of the display, in accordance with one embodiment of the presentdisclosure.

FIG. 3B illustrates an image shown on a display and including a fovealregion of high resolution, wherein the foveal region corresponds to alocation of the display towards which the user is directing his or hergaze, in accordance with one embodiment of the present disclosure.

FIG. 4A illustrates a graphics processor implementing a graphicspipeline configured for foveated rendering, in accordance with oneembodiment of the present disclosure.

FIG. 4B illustrates a graphics processor implementing a graphicspipeline configured for foveated rendering as implemented throughdeferred tile rendering, in accordance with one embodiment of thepresent disclosure.

FIG. 5A is a flow diagram illustrating steps in a method forimplementing a graphics pipeline configured for foveated rendering,wherein portions of an image in a foveal region are rendered with higherresolution, and portions of an image in a peripheral region are renderedwith lower resolution, in accordance with one embodiment of thedisclosure.

FIG. 5B is a flow diagram illustrating steps in a method forimplementing a graphics pipeline configured for foveated rendering asimplemented through tile deferred rendering, wherein light sources fortiles outside the foveal region are aggregated, in accordance with oneembodiment of the present disclosure.

FIG. 6A is an illustration of a plurality of light sources affecting ascene and a cluster of light sources forming an aggregated light source,in accordance with one embodiment of the disclosure.

FIG. 6B is an illustration of a display partitioned into a plurality oftiles, wherein rendering is performed on a tile-by-tile basis in adeferred rendering system, in accordance with one embodiment of thepresent disclosure.

FIG. 7 is a diagram illustrating components of a head-mounted display,in accordance with an embodiment of the disclosure.

FIG. 8 is a block diagram of a Game System, according to variousembodiments of the disclosure.

DETAILED DESCRIPTION

Although the following detailed description contains many specificdetails for the purposes of illustration, anyone of ordinary skill inthe art will appreciate that many variations and alterations to thefollowing details are within the scope of the present disclosure.Accordingly, the aspects of the present disclosure described below areset forth without any loss of generality to, and without imposinglimitations upon, the claims that follow this description.

Generally speaking, the various embodiments of the present disclosuredescribe a graphics processor of a video rendering system that isconfigured to perform foveated rendering, wherein portions of images ina foveal region may be rendered with high resolution and portionsoutside the foveal region may be rendered with lower resolution. Inparticular, image portions that are outside the foveal region arerendered with aggregated light sources in order to reduce thecomputation for rendering images, wherein an aggregated light sourceclosely approximates the total effect produced from each individuallight source of a corresponding cluster of light sources. In someembodiments, the foveated rendering is performed within or for thepurposes of displaying images within a head mounted display (HMD).

With the above general understanding of the various embodiments, exampledetails of the embodiments will now be described with reference to thevarious drawings.

Throughout the specification, the reference to “video game” or “gamingapplication” is meant to represent any type of interactive applicationthat is directed through execution of input commands For illustrationpurposes only, an interactive application includes applications forgaming, word processing, video processing, video game processing, etc.Further, the terms video game and gaming application areinterchangeable.

FIG. 1A illustrates a system for interactive gameplay of a video game,in accordance with an embodiment of the invention. A user 100 is shownwearing a head-mounted display (HMD) 102. The HMD 102 is worn in amanner similar to glasses, goggles, or a helmet, and is configured todisplay a video game from an interactive video game or other contentfrom interactive application, to the user 100. The HMD 102 provides avery immersive experience to the user by virtue of its provision ofdisplay mechanisms in close proximity to the user's eyes. Thus, the HMD102 can provide display regions to each of the user's eyes which occupylarge portions or even the entirety of the field of view of the user.Though FIGS. 1A-1B and in other figures are shown using HMDs fordisplaying rendered images, embodiments of the present invention arewell suited for performing foveated rendering in any display device,wherein the foveated rendering includes the aggregation of light sourcesfor rendering objects displayed in peripheral regions, and displayingthe rendered images on any display.

In one embodiment, HMD 102 is configurable to display images configuredwith foveated rendering, wherein portions of images in a foveal regionare displayed with high resolution and portions outside the fovealregion are displayed with lower resolution. In particular, imageportions that are outside the foveal region are rendered with aggregatedlight sources, wherein an aggregated light source closely approximatesthe total effect produced from each individual light source of acorresponding cluster of light sources. As such, instead of computingthe individual effects of each of the individual light sources of acluster, only the effect of the aggregated light source of the clusteris computed. In that manner, the total bandwidth for the sequence ofvideo frames being displayed is reduced.

In one embodiment, the HMD 102 can be connected to a computer or gamingconsole 106. The connection to computer 106 can be wired or wireless.The computer 106 can be any general or special purpose computer known inthe art, including but not limited to, a gaming console, personalcomputer, laptop, tablet computer, mobile device, cellular phone,tablet, thin client, set-top box, media streaming device, etc. In oneembodiment, the computer 106 can be configured to execute a video game,and output the video and audio from the video game for rendering by theHMD 102. The computer 106 is not restricted to executing a video gamebut may also be configured to execute an interactive application, whichoutputs VR content 191 for rendering by the HMD 102. In one embodiment,computer 106 performs the aggregating of light sources when renderingportions of images outside of the foveal region.

The user 100 may operate a controller 104 to provide input for the videogame. Additionally, a camera 108 can be configured to capture one ormore images of the interactive environment in which the user 100 islocated. These captured images can be analyzed to determine the locationand movements of the user 100, the HMD 102, and the controller 104. Inone embodiment, the controller 104 includes a light or other markerelements which can be tracked to determine its location and orientation.The camera 108 can include one or more microphones to capture sound fromthe interactive environment. Sound captured by a microphone array may beprocessed to identify the location of a sound source. Sound from anidentified location can be selectively utilized or processed to theexclusion of other sounds not from the identified location. Furthermore,the camera 108 can be defined to include multiple image capture devices(e.g. stereoscopic pair of cameras), an IR camera, a depth camera, andcombinations thereof.

In another embodiment, the computer 106 functions as a thin client incommunication over a network with a cloud gaming provider 112. The cloudgaming provider 112 maintains and executes the video game being playedby the user 102. The computer 106 transmits inputs from the HMD 102, thecontroller 104 and the camera 108, to the cloud gaming provider, whichprocesses the inputs to affect the game state of the executing videogame. The output from the executing video game, such as video data,audio data, and haptic feedback data, is transmitted to the computer106. The computer 106 may further process the data before transmissionor may directly transmit the data to the relevant devices. For example,video and audio streams are provided to the HMD 102, whereas the hapticfeedback data is used to generate a vibration feedback command, which isprovided to the controller 104.

In one embodiment, the HMD 102, controller 104, and camera 108, maythemselves be networked devices that connect to the network 110 tocommunicate with the cloud gaming provider 112. For example, thecomputer 106 may be a local network device, such as a router, that doesnot otherwise perform video game processing, but facilitates passagenetwork traffic. The connections to the network by the HMD 102,controller 104, and camera (i.e., image capture device) 108 may be wiredor wireless.

In yet another embodiment, the computer 106 may execute a portion of thevideo game, while the remaining portion of the video game may beexecuted on a cloud gaming provider 112. In other embodiments, portionsof the video game may also be executed on HMD 102. For example, arequest for downloading the video game from the computer 106 may beserviced by the cloud gaming provider 112. While the request is beingserviced, the cloud gaming provider 112 may execute a portion of thevideo game and provide game content to the computer 106 for rendering onthe HMD 102. The computer 106 may communicate with the cloud gamingprovider 112 over a network 110. Inputs received from the HMD 102, thecontroller 104 and the camera 108, are transmitted to the cloud gamingprovider 112, while the video game is downloading on to the computer106. The cloud gaming provider 112 processes the inputs to affect thegame state of the executing video game. The output from the executingvideo game, such as video data, audio data, and haptic feedback data, istransmitted to the computer 106 for onward transmission to therespective devices.

Once the video game has been completely downloaded to the computer 106,the computer 106 may execute the video game and resume game play of thevideo game from where it was left off on the cloud gaming provider 112.The inputs from the HMD 102, the controller 104, and the camera 108 areprocessed by the computer 106, and the game state of the video game isadjusted, in response to the inputs received from the HMD 102, thecontroller 104, and the camera 108. In such embodiments, a game state ofthe video game at the computer 106 is synchronized with the game stateat the cloud gaming provider 112. The synchronization may be doneperiodically to keep the state of the video game current at both thecomputer 106 and the cloud gaming provider 112. The computer 106 maydirectly transmit the output data to the relevant devices. For example,video and audio streams are provided to the HMD 102, whereas the hapticfeedback data is used to generate a vibration feedback command, which isprovided to the controller 104.

FIG. 1B conceptually illustrates the function of a HMD 102 inconjunction with the generation of VR content 191 (e.g., execution of anapplication and/or video game, etc.), in accordance with an embodimentof the invention. In some implementations, the VR content engine 120 isbeing executed on a computer 106 (not shown) that is communicativelycoupled to the HMD 102. The computer may be local to the HMD (e.g., partof local area network) or may be remotely located (e.g., part of a widearea network, a cloud network, etc.) and accessed via a network. Thecommunication between the HMD 102 and the computer 106 may follow awired or a wireless connection protocol. For example, the VR contentengine 120 executing an application may be a video game engine executinga video game, and is configured to receive inputs to update a game stateof the video game. The following description of FIG. 1B is describedwithin the context of the VR content engine 120 executing a video game,for purposes of brevity and clarity, and is intended to represent theexecution of any application capable of generating VR content 191. Thegame state of the video game can be defined, at least in part, by valuesof various parameters of the video game which define various aspects ofthe current gameplay, such as the presence and location of objects, theconditions of a virtual environment, the triggering of events, userprofiles, view perspectives, etc.

In the illustrated embodiment, the game engine 120 receives, by way ofexample, controller input 161, audio input 162 and motion input 163. Thecontroller input 161 may be defined from the operation of a gamingcontroller separate from the HMD 102, such as a hand-held gamingcontroller 104 (e.g. Sony DUALSHOCK®4 wireless controller, SonyPlayStation® Move motion controller) or wearable controllers, such aswearable glove interface controller, etc. By way of example, controllerinput 161 may include directional inputs, button presses, triggeractivation, movements, gestures or other kinds of inputs processed fromthe operation of a gaming controller. The audio input 162 can beprocessed from a microphone 151 of the HMD 102, or from a microphoneincluded in the image capture device 108 or elsewhere within the localsystem environment. The motion input 163 can be processed from a motionsensor 159 included in the HMD 102, or from image capture device 108 asit captures images of the HMD 102. The VR content engine 120 (e.g.,executing a gaming application) receives inputs which are processedaccording to the configuration of the game engine to update the gamestate of the video game. The engine 120 outputs game state data tovarious rendering modules which process the game state data to definecontent which will be presented to the user.

In the illustrated embodiment, a video rendering module 183 is definedto render a video stream for presentation on the HMD 102. Foveated viewrenderer 190 is configured to render foveated images in conjunction withand/or independent of video rendering module 183. Additionally, thefunctionality provided by the foveated view renderer 190 may beincorporated within the video rendering module 183, in embodiments. Inparticular, foveated view renderer 190 is configured to perform foveatedrendering, wherein portions of images in a foveal region with highresolution and portions outside the foveal region with lower resolution.In particular, image portions that are outside the foveal region arerendered with aggregated light sources in order to reduce thecomputation for rendering images, wherein an aggregated light sourceclosely approximates the total effect produced from each individuallight source of a corresponding cluster of light sources. Clustergenerator 192 is configured to aggregate light sources in a cluster oflight sources for purposes of foveated rendering. More particularly,deferred tile renderer 194 is configured to perform deferred tilerendering by rendering objects in individually tiled portions of pixelsof the display, wherein for tiles displaying objects located outside ofthe foveal region light sources are aggregated by the cluster generator192 when performing foveated rendering.

A lens of optics 170 in the HMD 102 is configured for viewing the VRcontent 191. A display screen 175 is disposed behind the lens of optics170, such that the lens of optics 170 is between the display screen 175and an eye 160 of the user, when the HMD 102 is worn by the user. Inthat manner, the video stream may be presented by the displayscreen/projector mechanism 175, and viewed through optics 170 by the eye160 of the user. An HMD user may elect to interact with the interactiveVR content 191 (e.g., VR video source, video game content, etc.) bywearing the HMD and selecting a video game for game play, for example.Interactive virtual reality (VR) scenes from the video game are renderedon the display screen 175 of the HMD. In that manner, the HMD allows theuser to completely immerse in the game play by provisioning displaymechanism of the HMD in close proximity to the user's eyes. The displayregions defined in the display screen of the HMD for rendering contentmay occupy large portions or even the entirety of the field of view ofthe user. Typically, each eye is supported by an associated lens ofoptics 170 which is viewing one or more display screens.

An audio rendering module 182 is configured to render an audio streamfor listening by the user. In one embodiment, the audio stream is outputthrough a speaker 152 associated with the HMD 102. It should beappreciated that speaker 152 may take the form of an open air speaker,headphones, or any other kind of speaker capable of presenting audio.

In one embodiment, a gaze tracking camera 165 is included in the HMD 102to enable tracking of the gaze of the user. Although only one gazetracking camera 165 is included, it should be noted that more than onegaze tracking camera may be employed to track the gaze of the user. Thegaze tracking camera captures images of the user's eyes, which areanalyzed to determine the gaze direction of the user. In one embodiment,information about the gaze direction of the user can be utilized toaffect the video rendering. For example, if a user's eyes are determinedto be looking in a specific direction, then the video rendering for thatdirection can be prioritized or emphasized, such as by providing greaterdetail, higher resolution through foveated rendering as provided byfoveated view renderer 190, or faster updates in the region where theuser is looking. It should be appreciated that the gaze direction of theuser can be defined relative to the head mounted display, relative to areal environment in which the user is situated, and/or relative to avirtual environment that is being rendered on the head mounted display.

Broadly speaking, analysis of images captured by the gaze trackingcamera 165, when considered alone, provides for a gaze direction of theuser relative to the HMD 102. However, when considered in combinationwith the tracked location and orientation of the HMD 102, a real-worldgaze direction of the user can be determined, as the location andorientation of the HMD 102 is synonymous with the location andorientation of the user's head. That is, the real-world gaze directionof the user can be determined from tracking the positional movements ofthe user's eyes and tracking the location and orientation of the HMD102. When a view of a virtual environment is rendered on the HMD 102,the real-world gaze direction of the user can be applied to determine avirtual world gaze direction of the user in the virtual environment.

Additionally, a tactile feedback module 181 is configured to providesignals to tactile feedback hardware included in either the HMD 102 oranother device operated by the HMD user, such as a controller 104. Thetactile feedback may take the form of various kinds of tactilesensations, such as vibration feedback, temperature feedback, pressurefeedback, etc.

FIG. 2A illustrates a system 200A providing gaming control to one ormore users playing one or more gaming applications that are executinglocally to the corresponding user, and wherein back-end server support(e.g., accessible through game server 205) may be configured to supporta plurality of local computing devices supporting a plurality of users,wherein each local computing device may be executing an instance of avideo game, such as in a single-player or multi-player video game. Forexample, in a multi-player mode, while the video game is executinglocally, the cloud game network concurrently receives information (e.g.,game state data) from each local computing device and distributes thatinformation accordingly throughout one or more of the local computingdevices so that each user is able to interact with other users (e.g.,through corresponding characters in the video game) in the gamingenvironment of the multi-player video game. In that manner, the cloudgame network coordinates and combines the game plays for each of theusers within the multi-player gaming environment. Referring now to thedrawings, like referenced numerals designate identical or correspondingparts.

As shown in FIG. 2A, a plurality of users 215 (e.g., user 100A, user100B . . . user 100N) is playing a plurality of gaming applications,wherein each of the gaming applications is executed locally on acorresponding client device 106 (e.g., game console) of a correspondinguser. Each of the client devices 106 may be configured similarly in thatlocal execution of a corresponding gaming application is performed. Forexample, user 100A may be playing a first gaming application on acorresponding client device 106, wherein an instance of the first gamingapplication is executed by a corresponding game title execution engine211. Game logic 226A (e.g., executable code) implementing the firstgaming application is stored on the corresponding client device 106, andis used to execute the first gaming application. For purposes ofillustration, game logic may be delivered to the corresponding clientdevice 106 through a portable medium (e.g., flash drive, compact disk,etc.) or through a network (e.g., downloaded through the internet 250from a gaming provider). In addition, user 100B is playing a secondgaming application on a corresponding client device 106, wherein aninstance of the second gaming application is executed by a correspondinggame title execution engine 211. The second gaming application may beidentical to the first gaming application executing for user 100A or adifferent gaming application. Game logic 226B (e.g., executable code)implementing the second gaming application is stored on thecorresponding client device 106 as previously described, and is used toexecute the second gaming application. Further, user 100N is playing anNth gaming application on a corresponding client device 106, wherein aninstance of the Nth gaming application is executed by a correspondinggame title execution engine 211. The Nth gaming application may beidentical to the first or second gaming application, or may be acompletely different gaming application. Game logic 226N (e.g.,executable code) implementing the third gaming application is stored onthe corresponding client device 106 as previously described, and is usedto execute the Nth gaming application.

As previously described, client device 106 may receive input fromvarious types of input devices, such as game controllers, tabletcomputers, keyboards, gestures captured by video cameras, mice touchpads, etc. Client device 106 can be any type of computing device havingat least a memory and a processor module that is capable of connectingto the game server 205 over network 150. Also, client device 106 of acorresponding user is configured for generating rendered images executedby the game title execution engine 211 executing locally or remotely,and for displaying the rendered images on a display (e.g., display 11,HMD 102, etc.). For example, the rendered images may be associated withan instance of the first gaming application executing on client device106 of user 100A. For example, a corresponding client device 106 isconfigured to interact with an instance of a corresponding gamingapplication as executed locally or remotely to implement a game play ofa corresponding user, such as through input commands that are used todrive game play. Some examples of client device 106 include a personalcomputer (PC), a game console, a home theater device, a general purposecomputer, mobile computing device, a tablet, a phone, or any other typesof computing devices that can interact with the game server 205 toexecute an instance of a video game.

In one embodiment, client device 106 is operating in a single-playermode for a corresponding user that is playing a gaming application. Inanother embodiment, multiple client devices 106 are operating in amulti-player mode for corresponding users that are each playing aspecific gaming application. In that case, back-end server support viathe game server may provide multi-player functionality, such as throughthe multi-player processing engine 219. In particular, multi-playerprocessing engine 219 is configured for controlling a multi-playergaming session for a particular gaming application. For example,multi-player processing engine 219 communicates with the multi-playersession controller 216, which is configured to establish and maintaincommunication sessions with each of the users and/or playersparticipating in the multi-player gaming session. In that manner, usersin the session can communicate with each other as controlled by themulti-player session controller 216.

Further, multi-player processing engine 219 communicates withmulti-player logic 218 in order to enable interaction between userswithin corresponding gaming environments of each user. In particular,state sharing module 217 is configured to manage states for each of theusers in the multi-player gaming session. For example, state data mayinclude game state data that defines the state of the game play (of agaming application) for a corresponding user at a particular point. Forexample, game state data may include game characters, game objects, gameobject attributes, game attributes, game object state, graphic overlays,etc. In that manner, game state data allows for the generation of thegaming environment that exists at the corresponding point in the gamingapplication. Game state data may also include the state of every deviceused for rendering the game play, such as states of CPU, GPU, memory,register values, program counter value, programmable DMA state, buffereddata for the DMA, audio chip state, CD-ROM state, etc. Game state datamay also identify which parts of the executable code need to be loadedto execute the video game from that point. Game state data may be storedin database (not shown), and is accessible by state sharing module 217.

Further, state data may include user saved data that includesinformation that personalizes the video game for the correspondingplayer. This includes information associated with the character playedby the user, so that the video game is rendered with a character thatmay be unique to that user (e.g., location, shape, look, clothing,weaponry, etc.). In that manner, the user saved data enables generationof a character for the game play of a corresponding user, wherein thecharacter has a state that corresponds to the point in the gamingapplication experienced currently by a corresponding user. For example,user saved data may include the game difficulty selected by acorresponding user 100A when playing the game, game level, characterattributes, character location, number of lives left, the total possiblenumber of lives available, armor, trophy, time counter values, etc. Usersaved data may also include user profile data that identifies acorresponding user 100A, for example. User saved data may be stored instorage (not shown).

In that manner, the multi-player processing engine 219 using the statesharing data 217 and multi-player logic 218 is able to overlay/insertobjects and characters into each of the gaming environments of the usersparticipating in the multi-player gaming session. For example, acharacter of a first user is overlaid/inserted into the gamingenvironment of a second user. This allows for interaction between usersin the multi-player gaming session via each of their respective gamingenvironments (e.g., as displayed on a screen).

FIG. 2B illustrates a system 200B providing gaming control to one ormore users 215 (e.g., users 100L, 100M . . . 100Z) playing a gamingapplication in respective VR viewing environments as executed over acloud game network, in accordance with one embodiment of the presentdisclosure. In some embodiments, the cloud game network may be a gamecloud system 210 that includes a plurality of virtual machines (VMs)running on a hypervisor of a host machine, with one or more virtualmachines configured to execute a game processor module utilizing thehardware resources available to the hypervisor of the host. Referringnow to the drawings, like referenced numerals designate identical orcorresponding parts.

As shown, the game cloud system 210 includes a game server 205 thatprovides access to a plurality of interactive video games or gamingapplications. Game server 205 may be any type of server computing deviceavailable in the cloud, and may be configured as one or more virtualmachines executing on one or more hosts. For example, game server 205may manage a virtual machine supporting a game processor thatinstantiates an instance of a gaming application for a user. As such, aplurality of game processors of game server 205 associated with aplurality of virtual machines is configured to execute multipleinstances of the gaming application associated with game plays of theplurality of users 215. In that manner, back-end server support providesstreaming of media (e.g., video, audio, etc.) of game plays of aplurality of gaming applications to a plurality of corresponding users.

A plurality of users 215 accesses the game cloud system 210 via network250, wherein users (e.g., users 100L, 100M . . . 100Z) access network250 via corresponding client devices 106′, wherein client device 106′may be configured similarly as client device 106 of FIG. 2A (e.g.,including game executing engine 211, etc.), or may be configured as athin client providing that interfaces with a back end server providingcomputational functionality (e.g., including game executing engine 211).

In particular, a client device 106′ of a corresponding user 100L isconfigured for requesting access to gaming applications over a network250, such as the internet, and for rendering instances of gamingapplication (e.g., video game) executed by the game server 205 anddelivered to a display device associated with the corresponding user100L. For example, user 100L may be interacting through client device106′ with an instance of a gaming application executing on gameprocessor of game server 205. More particularly, an instance of thegaming application is executed by the game title execution engine 211.Game logic (e.g., executable code) implementing the gaming applicationis stored and accessible through a data store (not shown), and is usedto execute the gaming application. Game title processing engine 211 isable to support a plurality of gaming applications using a plurality ofgame logics 277, as shown.

As previously described, client device 106′ may receive input fromvarious types of input devices, such as game controllers, tabletcomputers, keyboards, gestures captured by video cameras, mice touchpads, etc. Client device 106′ can be any type of computing device havingat least a memory and a processor module that is capable of connectingto the game server 205 over network 250. Also, client device 106′ of acorresponding user is configured for generating rendered images executedby the game title execution engine 211 executing locally or remotely,and for displaying the rendered images on a display. For example, therendered images may be associated with an instance of the first gamingapplication executing on client device 106′ of user 100L. For example, acorresponding client device 106′ is configured to interact with aninstance of a corresponding gaming application as executed locally orremotely to implement a game play of a corresponding user, such asthrough input commands that are used to drive game play.

Client device 106′ is configured for receiving rendered images, and fordisplaying the rendered images on display 11 and/or HMD 102 (e.g.,displaying VR content). For example, through cloud based services therendered images may be delivered by an instance of a gaming applicationexecuting on game executing engine 211 of game server 205 in associationwith user 100. In another example, through local game processing, therendered images may be delivered by the local game executing engine 211.In either case, client device 106 is configured to interact with thelocal or remote executing engine 211 in association with the game playof a corresponding user 100, such as through input commands that areused to drive game play. In another implementation, the rendered imagesmay be streamed to a smartphone or tablet, wirelessly or wired, directfrom the cloud based services or via the client device 106 (e.g.,PlayStation® Remote Play).

In another embodiment, multi-player processing engine 219, previouslydescribed, provides for controlling a multi-player gaming session for agaming application. In particular, when the multi-player processingengine 219 is managing the multi-player gaming session, the multi-playersession controller 216 is configured to establish and maintaincommunication sessions with each of the users and/or players in themulti-player session. In that manner, users in the session cancommunicate with each other as controlled by the multi-player sessioncontroller 216.

Further, multi-player processing engine 219 communicates withmulti-player logic 218 in order to enable interaction between userswithin corresponding gaming environments of each user. In particular,state sharing module 217 is configured to manage states for each of theusers in the multi-player gaming session. For example, state data mayinclude game state data that defines the state of the game play (of agaming application) for a corresponding user 100 at a particular point,as previously described. Further, state data may include user saved datathat includes information that personalizes the video game for thecorresponding player, as previously described. For example, state dataincludes information associated with the user's character, so that thevideo game is rendered with a character that may be unique to that user(e.g., shape, look, clothing, weaponry, etc.). In that manner, themulti-player processing engine 219 using the state sharing data 217 andmulti-player logic 218 is able to overlay/insert objects and charactersinto each of the gaming environments of the users participating in themulti-player gaming session. This allows for interaction between usersin the multi-player gaming session via each of their respective gamingenvironments (e.g., as displayed on a screen).

FIG. 3A illustrates an image 310 shown on a display 300, wherein theimage includes a foveal region 310A of high resolution, wherein thefoveal region corresponds to a center of the display, in accordance withone embodiment of the present disclosure. In particular, image 310includes rows and columns of the letter “E” for simplicity and clarity.The image 310 is partitioned into multiple regions, including a fovealregion 310A and a peripheral region 310B.

As shown, the foveal region 310A is static and corresponds to the centerof the display 300. The foveal region 310A is assumed to be the regiontowards which the user mostly directs his or her gaze (e.g., using thefovea of the eye), such as when viewing graphics of a video game. Thoughthe gaze of the user may occasionally be directed off center, the gazeis mostly directed to the center (to view the main content). In somecases, the image is designed to initially bring the gaze of the useroff-center (e.g., to view an object of interest), but then to bring thegaze back to the center (e.g., by moving the object towards the fovealregion 310A).

In particular, the portion or portions of any image, such as image 310,as displayed and located within the foveal region 310A will be renderedat higher resolution. For example, the graphics pipeline will renderportions of the image in the foveal region 310A while minimizing the useof any techniques used to reduce computational complexity. Inparticular, for embodiments of the present invention, light sourcesaffecting objects displayed using pixels corresponding to the fovealregion 310A are individually computed within the graphics pipeline inorder to determine each of their effects on the objects (e.g., color,texture, shadowing, etc. on polygons of the objects). Representative ofthe higher resolution, the letter “E” objects as displayed within thefoveal region 310A are shown with clarity, vibrant color, and minimalblurriness. This is consistent with and takes advantage of the gaze ofthe user being directed towards the foveal region 310A on display 300.

In addition, the portion or portions of an image, such as image 310, asdisposed and located in the peripheral region 310B will be rendered atlower resolution (e.g., lower than the resolution of the portions of theimage and/or objects located in the foveal region 310A). The gaze of theuser is not typically directed to the objects located in and/ordisplayed in the peripheral region 310B, as the main focus of the gazeis directed to the objects in the foveal region 310A. Consistent withreal-life views into a scene, the objects in the peripheral region 310Bare rendered with lower resolution, with enough detail so that the useris able to perceive moving objects (e.g., a human walkingstraight-legged instead of with knees bent) and with sufficient contrastwithin an object or between objects in the peripheral region 310B, forexample. To achieve rendering at lower resolutions, the graphicspipeline may render portions of the image in the peripheral region 310Busing computationally efficient techniques that reduce computationalcomplexity. In particular, for embodiments of the present invention,light sources affecting objects displayed using pixels corresponding topixels in the peripheral region 310B may be grouped into one or moreclusters, and wherein light sources in a cluster are aggregated into asingle light source that closely approximates the total effect producedfrom each individual light source of the corresponding cluster of lightsources. As such, instead of computing the individual effects of each ofthe individual light sources of a cluster, only the effect of theaggregated light source of the cluster is computed. This reducescomputational processing when rendering the overall image, andespecially for objects rendered in the peripheral region 310B.

FIG. 3B illustrates an image 310′ shown on a display 300 and including afoveal region 310A′ of high resolution, wherein the foveal regioncorresponds to a location of the display towards which the user isdirecting his or her gaze, in accordance with one embodiment of thepresent disclosure. In particular, image 310′ is similar to image 310shown in FIG. 3A and includes rows and columns of the letter “E” forsimplicity and clarity. The image 310 is partitioned into multipleregions, including a foveal region 310A′ and a peripheral region 310B′.

As shown, the foveal region 310A′ is dynamically moving throughoutdisplay 300 depending on which direction the gaze of the user isdirected towards. As previously described, the gaze may be tracked usinggaze tracking camera 165 of HMD 102, for example. As such, the fovealregion 310A′ may not necessarily correspond to the center of display300, but instead correlates to the actual direction and focus ofattention of the user within image 310′. That is, the foveal region310A′ dynamically moves with the movement of the eye and/or eyes of theuser.

As previously introduced, the portion or portions of any image, such asimage 310′, as displayed and located within the foveal region 310A′ willbe rendered at higher resolution by minimizing the use of any renderingtechniques used to reduce computational complexity when renderingobjects located in the foveal region 310A′. In particular, forembodiments of the present invention, light sources affecting objectsdisplayed using pixels corresponding to the foveal region 310A′ areindividually computed within the graphics pipeline in order to determineeach of their effects on the objects (e.g., color, texture, shadowing,etc. on polygons of the objects). Representative of the higherresolution, the letter “E” objects as displayed within the foveal region310A′ are shown with clarity, vibrant color, and minimal blurriness.

In addition, the portion or portions of an image, such as image 310′, asdisposed and located in the peripheral region 310B′ will be rendered atlower resolution (e.g., lower than the resolution of the portions of theimage and/or objects located in the foveal region 310A). As previouslyintroduced, the gaze of the user is not typically directed to theobjects located in and/or displayed in the peripheral region 310B′, asthe main focus of the gaze is directed to the objects in the fovealregion 310A′. As such, the objects in the peripheral region 310B′ arerendered with lower resolution, with enough detail so that the user isable to perceive moving objects (e.g., a human walking straight-leggedinstead of with knees bent) and with sufficient contrast within anobject or between objects in the peripheral region 310B′, for example.To achieve rendering at lower resolutions, the graphics pipeline mayrender portions of the image in the peripheral region 310B′ usingcomputationally efficient techniques that reduce computationalcomplexity. In particular, for embodiments of the present invention,light sources affecting objects displayed using pixels corresponding topixels in the peripheral region 310B′ may be grouped into one or moreclusters, and wherein light sources in a cluster are aggregated into asingle light source that closely approximates the total effect producedfrom each individual light source of the corresponding cluster of lightsources to reduce computational processing when rendering the overallimage, and especially for objects in the peripheral region 310B′.

FIG. 4A illustrates a graphics processor implementing a graphicspipeline 400A configured for foveated rendering, in accordance with oneembodiment of the present disclosure. The graphics pipeline 400A isillustrative of the general process for rendering images using 3D (threedimensional) polygon rendering processes, but includes an additionalprogrammable element within the pipeline that performs foveatedrendering. The graphics pipeline 400A for a rendered image outputscorresponding color information for each of the pixels in a display,wherein the color information may represent texture and shading (e.g.,color, shadowing, etc.). Graphics pipeline 400A is implementable withinthe game console 106 of FIG. 1A, VR content engine 120 of FIG. 1B,client devices 106 of FIGS. 2A and 2B, and/or game title processingengine 211 of FIG. 2B.

As shown, the graphics pipeline receives input geometries 405. Forexample, the input geometries 405 may include vertices within a 3Dgaming world, and information corresponding to each of the vertices. Agiven object within the gaming world can be represented using polygons(e.g., triangles) defined by vertices, wherein the surface of acorresponding polygon is then processed through the graphics pipeline400A to achieve a final effect (e.g., color, texture, etc.). Vertexattributes may include normal (e.g., which direction is the light inrelation to the vertex), color (e.g., RGB—red, green, and blue triple,etc.), and texture coordinate/mapping information.

The vertex shader and/or program 410 receives the input geometries 405,and builds the polygons or primitives that make up the objects withinthe 3D scene. That is, the vertex shader 410 builds up the objects usingthe primitives as they are placed within the gaming world. The vertexshader 410 may be configured to perform lighting and shadowingcalculations for the polygons, which is dependent on the lighting forthe scene. The primitives are output by the vertex shader 410 anddelivered to the next stage of the graphics pipeline 400A. Additionaloperations may also be performed by the vertex shader 410 such asclipping (e.g., identify and disregard primitives that are outside theviewing frustum as defined by the viewing location in the gaming world).

The primitives are fed into the rasterizer 420 that is configured toproject objects in the scene to a two-dimensional (2D) image planedefined by the viewing location in the 3D gaming world (e.g., cameralocation, user eye location, etc.). At a simplistic level, therasterizer 420 looks at each primitive and determines which pixels areaffected by the corresponding primitive. In particular, the rasterizer420 partitions the primitives into pixel sized fragments, wherein eachfragment corresponds to a pixel in the display. It is important to notethat one or more fragments may contribute to the color of acorresponding pixel when displaying an image. Additional operations mayalso be performed by the rasterizer 420 such as clipping (identify anddisregard fragments that are outside the viewing frustum) and culling(disregard fragments that are occluded by closer objects) to the viewinglocation.

The foveated fragment shader and/or program 430 at its core performsshading operations on the fragments to determine how the color andbrightness of a primitive varies with available lighting. For example,fragment shader 430 may determine depth, color, normal and texturecoordinates (e.g., texture details) for each fragment, and may furtherdetermine appropriate levels of light, darkness, and color for thefragments. In particular, fragment shader 430 calculates the traits ofeach fragment, including color and other attributes (e.g., z-depth fordistance from the viewing location, and alpha values for transparency).In addition, the fragment shader 430 applies lighting effects to thefragments based on the available lighting affecting the correspondingfragments. Further, the fragment shader 430 may apply shadowing effectsfor each fragment. For purposes of description only light sources aredescribed as being point lights having a definite position in the gamingworld, and which radiates light omni-directionally. Other lighting isavailable, such as directional lighting, etc.

More particularly, the foveated fragment shader 430 performs shadingoperations as described above based on whether the fragment is withinthe foveal region or peripheral region. Fragments that are locatedwithin the foveal region of the displayed image are processed usingshading operations at high resolution, without regard to processingefficiency in order to achieve detailed texture and color values forfragments within the foveal region. On the other hand, the foveatedfragment shader 430 performs shading operations on fragments that arelocated within the peripheral region with an interest in processingefficiency in order to process fragments with sufficient detail withminimal operations, such as providing movement and sufficient contrast.For example, in embodiments of the present invention light sourcesaffecting fragments displayed in pixels in the peripheral region may begrouped into one or more clusters, and wherein light sources in acluster are aggregated into a single light source that closelyapproximates the total effect produced from each individual light sourceof the corresponding cluster of light sources to reduce computationalprocessing when rendering the overall image. The output of the fragmentshader 430 includes processed fragments (e.g., texture and shadinginformation) and is delivered to the next stage of the graphics pipeline400A.

The output merging component 440 calculates the traits of each pixeldepending on the fragments that contribute and/or affect eachcorresponding pixel. That is, the fragments of all primitives in the 3Dgaming world are combined into the 2D color pixel for the display. Forexample, fragments that contribute to texture and shading informationfor a corresponding pixel are combined to output a final color value forthe pixel delivered to the next stage in the graphics pipeline 400A. Theoutput merging component 440 may perform optional blending of valuesbetween fragments and/or pixels determined from the fragment shader 430.

Color values for each pixel in the display are stored in the framebuffer 455. These values are scanned to the corresponding pixels whendisplaying a corresponding image of the scene. In particular, thedisplay reads color values from the frame buffer for each pixel,row-by-row, from left-to-right or right-to-left, top-to-bottom orbottom-to-top, or any other pattern, and illuminates pixels using thosepixel values when displaying the image.

FIG. 4B illustrates a graphics processor implementing a graphicspipeline configured for foveated rendering as implemented throughdeferred tile rendering, in accordance with one embodiment of thepresent disclosure. The graphics pipeline 400B for a rendered imageoutputs corresponding color information for each of the pixels in adisplay, wherein the color information may represent texture and shading(e.g., color, shadowing, etc.). Graphics pipeline 400B is implementablewithin the game console 106 of FIG. 1A, VR content engine 120 of FIG.1B, client devices 106 of FIGS. 2A and 2B, and/or game title processingengine 211 of FIG. 2B.

Deferred rendering may be performed in any number of ways, and isdescribed generally as implemented through graphics pipeline 400B. Ingeneral, deferred tile rendering processes each image of a scene of agaming world as multiple sub-images. Each sub-image that is generatedcorresponds to a tile of the display, wherein the display is partitionedinto multiple tiles. That is, each tile is rendered separately and theimage is fully rendered when all sub-images associated with the varioustiles have been rendered. Associating objects with particular tiles maybe performed at one or more stages of the graphics pipeline 400B. Asshown, in one embodiment tile generator 460 may perform tiling of theimage between the vertex shader 410 and the rasterizer 420. In anotherembodiment tile generator 460′ may perform tiling of the image betweenthe rasterizer 420 and the fragment shader 430′.

Pipeline 400B is similar to pipeline 400A, wherein like referencednumerals designate identical or corresponding parts, and is used torender images using a 3D polygon rendering processes, and includesadditional programmable elements that perform foveated rendering. Inparticular, pipeline 400B is configured to perform tiled rendering, alsoknown as deferred tile rendering, when rendering images. Each of thetiles render their portion of the image with foveated renderingprocesses, including rendering image portions that are outside a fovealregion with lower resolution. For example, image portions in theperipheral region are rendered using an aggregated light source thatclosely approximates the total effect produced from individual lightsources of a corresponding cluster of light sources.

As previously introduced, graphics pipeline 400B receives inputgeometries 405 that may include vertices within a gaming world, andinformation corresponding to each of the vertices. Vertex attributes mayinclude normal, color, and texture coordinate/mapping information. Thevertices define objects within the gaming world, and more particularlydefine polygons (e.g., triangles) that form the surfaces of objectswithin the gaming world. The polygon is processed through graphicspipeline 400B to achieve a final effect (e.g., color, texture, etc.).

The vertex shader 410 receives the input geometries 405 and builds thepolygons or primitives that make up the objects within the 3D gamingworld. The primitives are output by the vertex shader and delivered tothe next stage of the graphics pipeline. Additional operations may beperformed by the vertex shader, such as lighting and shadowingcalculations, clipping, and/or culling.

In one embodiment, the tile generator 460 is configured to determinewhere a primitive is ultimately displayed on a tile of a display. Thedisplay is partitioned into a plurality of tiles, each tile beingassociated with a corresponding number of unique pixels. In particular,the tile generator 460 is configured to sort the primitives to determinewhich primitives overlap or are associated with a corresponding tile.Once the primitives are sorted, the primitives can be processed on atile-by-tile basis through the graphics pipeline 400B.

The primitives are then fed into the rasterizer 420 that is configuredto project objects in the scene to a 2D image plane defined by a viewinglocation in the gaming world. If the primitives have been sorted bytile, then the rasterizer 420 is projecting objects onto a tile of the2D image plane. As previously introduced, the rasterizer 420 partitionsthe primitives into pixel sized fragments, wherein each fragmentcorresponds to a pixel in the display, and wherein one or more fragmentscontribute to the color of the pixel. Additional operations may also beperformed by the rasterizer 420 such as clipping and culling.

In another embodiment, the tile generator 460′ is configured todetermine where a fragment is ultimately displayed on a tile of adisplay based on the information determined by the rasterizer 420. Assuch, the tile generator 460′ is configured to sort the fragments todetermine which fragments overlap or are associated with a correspondingtile. Once the fragments are sorted, the fragments can be processed on atile-by-tile basis through the graphics pipeline 400B.

The tiled foveated fragment shader 430′ at its core performs shadingoperations on the fragments on a tile-by-tile basis to determine how thecolor and brightness of a primitive varies with available lighting. Forexample, fragment shader 430′ may determine depth, color, normal andtexture coordinates (e.g., texture details) for each fragment of acorresponding tile, and may further determine appropriate levels oflight, darkness, and color for the fragments. In particular, fragmentshader 430′ calculates the traits of each fragment, including color andother attributes (e.g., z-depth for distance from the viewing location,and alpha values for transparency). In addition, the fragment shader430′ applies lighting effects to the fragments based on the availablelighting affecting the corresponding fragments. Further, the fragmentshader 430′ may apply shadowing effects for each fragment. For purposesof description only light sources are described as being point lightshaving a definite position in the gaming world, and which radiates lightomni-directionally. Other lighting is available, such as directionallighting, etc.

More particularly, the foveated fragment shader 430′ performs shadingoperations as described above based on whether the fragment is withinthe foveal region or peripheral region. This may be determined initiallyby determining whether the tile associated with the fragment is withinthe foveal region nor peripheral region. Fragments that are locatedwithin the foveal region of the displayed image are processed usingshading operations at high resolution. On the other hand, the foveatedfragment shader 430′ performs shading operations on fragments that arelocated within the peripheral region with an interest in processingefficiency in order to process fragments with sufficient detail withminimal operations, such as providing movement and sufficient contrast.For example, in embodiments of the present invention light sourcesaffecting fragments of a particular tile that is located in theperipheral region may be grouped into one or more clusters, and whereinlight sources in a cluster are aggregated into a single light sourcethat closely approximates the total effect produced from each individuallight source of the corresponding cluster of light sources to reducecomputational processing when rendering the overall image. The output ofthe fragment shader 430′ includes processed fragments (e.g., texture andshading information) and is delivered to the next stage of the graphicspipeline 400B.

The tiled output merging component 440′ calculates the traits of eachpixel depending on the fragments that contribute and/or affect eachcorresponding pixel of a corresponding tile that is being processed.That is, the fragments of all primitives in the 3D gaming world arecombined into the 2D color pixel of a tile for the display. For example,fragments that contribute to texture and shading information for acorresponding pixel are combined to output a final color value for thepixel delivered to the next stage in the graphics pipeline 400B. Theoutput merging component 440′ may perform optional blending of valuesbetween fragments and/or pixels determined from the fragment shader430′.

Color values for each pixel in a tile of a display are stored in thetiled frame buffer 450′. These values are scanned to the correspondingpixels when displaying a corresponding sub-image corresponding to a tileof the scene. In one implementation, the frame buffer 450′ fills withinformation from all the tiles before drawing the image on the display.In particular, the display reads color values from the frame buffer foreach pixel, row-by-row, from least-to-right, top-to-bottom, andilluminates pixels using those pixel values when displaying the image.

With the detailed description of the various modules of the gamingserver and client device communicating over a network, flow diagram 500Aof FIG. 5A discloses a method for implementing a graphics pipelineconfigured to perform foveated rendering, wherein image portions thatare outside of a foveal region are rendered with one or more aggregatedlight sources, in accordance with one embodiment of the presentdisclosure. Flow diagram 500A is implemented within a client device orcloud based gaming system when rendering images, as previouslydescribed. In one implementation, the client device is a gaming console(also referred to as game console).

At 510, the method includes determining a plurality of light sourcesaffecting a virtual scene. The light sources affect objects within thescene, as processed by the graphics processor. The lighting from thelight sources will determine the final color of corresponding pixelswhen displaying a corresponding object within the scene as viewed from aviewing location.

At 515, the method includes projecting geometries of objects of an imageof the scene onto a plurality of pixels of a display from a firstpoint-of-view. As previously introduced, the rasterizer and/or vertexshader are configured to associate primitives and/or fragments ofprimitives with corresponding pixels. Specifically, the vertex shadermay be configured to associate primitives with corresponding pixels,wherein before rasterization the scene by primitives is partitioned intotiles. In addition, the rasterizer may be configured to associatefragments of primitives with corresponding pixels, wherein the scene byfragments may be partitioned into tiles.

At 520, the method includes partitioning a plurality of pixels of thedisplay into a plurality of tiles. In that manner, the graphicsprocessor is able to perform tile rendering or deferred tile renderingwhen processing the image. Further, a list of primitives that visiblyoverlap a corresponding tile from the viewing location may be createdfor each tile. In addition, a list of fragments that visibly overlap acorresponding tile from the viewing location may be created for eachtile. In that manner, tile-by-tile rendering can be performed.

At 525, the method includes defining a foveal region of highestresolution for the image as displayed. That is, the image is partitionedinto regions where the attention of the user is focused, either byassumption (e.g., generally at the center of the display to cover astatic area of the display) or by eye tracking (e.g., to determine thedirection of the gaze). The region where the attention of the user isfocused is the foveal region, and the portion of the image (e.g.,primitives and/or fragments) in the foveal region is rendered by thegraphics processor at high resolution. On the other hand, the peripheralregion does not have the attention of the user and the portion of theimage (e.g., primitives and/or fragments) in the peripheral region isrendered by the graphics processor at a lower resolution. Moreparticularly, a first subset of pixels is assigned to the foveal region,and wherein primitives and/or fragments associated with the first subsetis processed at high resolution. A second subset of pixels is assignedto a peripheral region that is outside of the foveal region, and whereinprimitives and/or fragments associated with the second subset isprocessed at the lower resolution (e.g., lower than the resolutionassociated with the foveal region).

When performing tiled rendering or deferred tile rendering, the methodat 530 includes determining a first set of light sources from aplurality of light sources that affect one or more objects displayed ina first tile that is in the peripheral region. When performing tiledrendering, light sources that affect each tile in association with theimage are determined. The light sources are taken from a plurality oflight sources that affect a scene. This reduces processing, as lightsources that do not affect objects in a tile are not considered whenrendering the sub-image associated with the tile. That is, for eachtile, a list of light sources affecting objects as drawn in thecorresponding tile is created. As such, for a given tile (e.g., thefirst tile) the first set of light sources is included in the listassociated with the first tile, wherein the first tile is in theperipheral region. That is, the objects displayed in the first tile arein the peripheral region, and as such, may be rendered at the lowerresolution. For example, rendering at a lower resolution may includeaggregating light sources as described below.

At 535, the method includes clustering at least two light sources fromthe first set into a first aggregated light source affecting the firsttile when rendering the image in pixels of the first tile. That is, whenperforming foveated rendering, the objects in the first tile aredetermined to be in the peripheral region, and those objects may berendered at lower resolution to include aggregating light sources. Thefirst set of light sources has previously been determined to affectobjects associated with or to be displayed in the first tile. Theselight sources may be aggregated into a smaller subset of light sources,such that rather than rendering the objects in the tile by consideringeach of the light sources in the first set on an individual basis, onlythe aggregated light sources are considered. Each of the aggregatedlight sources closely approximates the total effect produced by thefirst set of light sources on an individual basis. In that manner, lessprocessing is performed to render the objects of the tile.

FIG. 5B is a flow diagram 500B illustrating steps in a method forimplementing a graphics pipeline configured for foveated rendering asimplemented through tile deferred rendering, wherein light sources fortiles outside the foveal region are aggregated, in accordance with oneembodiment of the present disclosure. In particular, flow diagram 500Bis performed to cluster light sources on a tile-by-tile basis, whereinan aggregated light source represents a corresponding cluster of lightsources for purposes of rendering the objects of a correspond tile. Flowdiagram 500B is implemented within a client device or cloud based gamingsystem when rendering images, as previously described. In oneimplementation, the client device is a gaming console (also referred toas game console).

At 550, the method includes determining a target number of light sourcesfor a given tile. The target number corresponds to the total number ofaggregated and/or non-aggregated light sources that will be processedwhen rendering objects associated with that tile. For instance, themaximum number of average lights of the peripheral region is specifiedto constrain the clustering output. That is, the method includesconstraining the number of aggregated light sources affecting the firsttile to the target number

At 555, the method includes determining the set of light sources thataffect the given tile, wherein the set is taken from a plurality oflight sources that affect the scene. The target number of light sourcesis determined from the set of light sources that affect the tile,wherein the target number is less than or equal to the number in theset. The target number determines the maximum number of clusters oflight sources for the tile when performing rendering.

At 560, the method includes determining the number of light sources ineach of the clusters based on the target number and the set of lightsources. In one implementation, the number of light sources in a clusteris determined by dividing the set of light sources by the target number.For example, if there are 30 light sources in the set of light sources(e.g., M=30) affecting the tile, and the target number for the tile is5, then the number of light sources in each cluster is 6 (e.g.,M/[target number of lights]=6).

Operations 570 are performed in order to cluster the M lights into asmaller number of average light sources based on the relative distancebetween the light sources. In embodiments, to accomplish this distancebased aggregation (e.g., clustering), the clusters are generated usingnearest neighbor search or a k-means clustering techniques to choose theoptimal clustering of light sources, wherein each cluster of lightsources is associated with a corresponding aggregated light source, andwherein each cluster of light sources incudes at least one light source.

In one embodiment, a threshold is placed on the maximum distance forlights to be averaged. That is, the maximum distance between two lightsources in a cluster is constrained to a maximum distance. This willprevent averaging lights that are very far apart where the new averagelight position may be drastically different than the original lightpositions.

In particular, a looped process is performed for each light source (L_i)in the set of light sources (M) affecting the tile. For instance, at572, the method includes for a current light source (L_i), computingdistances between the current light source (L_i) and every light source(M) affecting the tile. At 574, the method includes for the currentlight source (L_i) in the set (M), determining a current cluster oflights (Cluster_i) which includes the current light source (L_i) andfive other light sources having the least distances to the current lightsource. At 576, the method includes determining a cluster error(Cluster_error_i) for the current cluster of light sources (Cluster_i).In one implementation, the cluster error (Cluster_error_i) is determinedby the sum of the distance between the current light source (L_i) andeach other light in the current cluster (Cluster_i). This process isperformed for each light L_i in the M lights, as determined by decisionstep 578.

At 580, the method includes choosing clusters having the least amount ofcluster error. That is, the method includes minimizing a plurality ofcluster errors for the plurality of clusters of light sources.Continuing with the example where M=30 light clusters, 6 clusters arechosen having the 6 lowest cluster errors (also considering anyconflicts of cluster errors). That is, for each cluster of light sources(cluster_i), a new aggregate light source is created, wherein theaggregated light source closely approximates the total effect producedfrom each individual light source of the cluster.

In one embodiment, to further define the cluster error, the clustererror corresponding to each cluster in the plurality of clusters oflight sources includes an iterative sum of the distances between acurrent light source and each of the other light sources, wherein thecurrent light source comprises a different light source in the clusterfor each iteration.

In one embodiment, the aggregated light source has characteristics thatare averages of the characteristics for the light sources in thecluster. For example, the aggregated light source has characteristicsthat are averages of distance, color, etc. The new intensity of theaggregated light source may be the sum of intensities for each lightsource in the cluster. Intensity may not be averages as this may lead toenergy loss affecting the scene. In particular, the aggregated lightsource may include a new location that is the average location of thelight sources in the cluster. A new color for the aggregated lightsource may be an average of the colors for each light source in thecluster.

In another implementation, the new color is the weighted sum of colorsfor each light source in the cluster. The weight for each light source(L_i) is determined based on intensities of a current light source (L_i)divided by the new intensity of the aggregated light source (e.g.,[Intensity_L_i]/[New Intensity]). As such, the new color is determinedas follows: [(Intensity_L_i/New Intensity)*(color_L_i)]. As such, theaggregated light source has characteristics that roughly maintain theoriginal intensity of each of the lights and will preserve the averagecolor of the lights by weighting the color relative to the totalintensity.

In one embodiment, the tile size is dynamically adjusted to fit thetarget number of clusters. In particular, the number of aggregated lightsources affecting a given tile is constrained to the target number. Whenit is determined that the set of light sources affecting the tile isless than the target number, the size of the given tile is expanded toaccommodate for the target number.

FIG. 6A is an illustration of a plurality of light sources affecting ascene 600A of a gaming world and at least one cluster of light sourcesforming an aggregated light source, in accordance with one embodiment ofthe disclosure. As shown, the gaming world 600 is defined by acoordinate system 601 in three dimensions. For purposes of simplicityand clarity one object 620 is shown in the scene 600A of the gamingworld, wherein the object 620 has a location within the gaming world.The scene 600A may be rendered from the viewpoint of an eye 630 of auser, wherein the viewpoint may be rendered in an image planecorresponding to a virtual display seen by eye 630.

Two clusters of light sources are shown affecting the scene 600A. Afirst cluster 615 a includes a plurality of light sources. Light sourcesin the first cluster 615 a are aggregated using techniques previouslydescribed into an aggregated light source 610 a. The aggregated lightsource 610 a has characteristics that closely approximate the totaleffect produced from each individual light source in the cluster 615 a.As shown, the aggregated light source 610 a has a location in the gamingworld that is defined by grid 640 a contained within the coordinatesystem 601. In addition, a second cluster 615 b includes a plurality oflight sources. Light sources in the second cluster 615 b are aggregatedusing techniques previously described into an aggregated light source610 b. The aggregated light source 610 b has characteristics thatclosely approximate the total effect produced from each individual lightsource in the cluster 615 b. As shown, the aggregated light source 610 bhas a location in the gaming world that is defined by grid 640 bcontained within the coordinate system 601.

FIG. 6B is an illustration of a display 650 that is partitioned into aplurality of tiles 660, wherein rendering is performed on a tile-by-tilebasis in a deferred rendering system of a graphics pipeline, inaccordance with one embodiment of the present disclosure. In particular,the plurality of tiles in display 650 may be defined by a rectangularpattern (as identified by rows and columns of tiles). Though the tilesare rectangular in shape, any shape may be used in any arrangement(e.g., rectangular, circular, spherical etc. within 2D, 3D, or moredimensions.

As shown, display 650 includes a foveal region 670 that displaysportions of an image that is rendered at higher resolution. Each of thetiles in the foveal region 670 includes a list of light sources thataffect the corresponding tile. For example, the tiles in the fovealregion 670 may be affected by varying numbers of light sources rangingfrom 10 to 35 light sources, wherein the plurality of light sourcesaffecting the scene may be equal to or greater than 35. For instance,objects as displayed in tile 660 p (e.g., in row 3 and col. 5) areaffected by 10 light sources; objects as displayed in tile 660 q (e.g.,in row 4 and col. 5) are affected by 14 light sources; and objects asdisplayed in tile 660 r (e.g., in row 3 and col. 7) are affected by 35light sources. Because the tiles are within the foveal region 670, thelight sources affecting a given tile are fully processed (e.g.,individually considered through the graphics pipeline) when renderingobjects (e.g., associated primitives and/or fragments) in that tile.

In addition, display 650 includes a peripheral region 675 that displaysportions of an image that is rendered at a lower resolution (e.g., lowerthan the portions of the image rendered in the foveal region 670). Eachof the tiles in the peripheral region 675 includes a list of lightsources that affect corresponding tile. Further, the tiles in theperipheral region 675 are rendered at lower resolution to include usingaggregated light sources for each of the tiles. For example, tile 660 nhas 25 light sources that affect objects drawn within that tile, whichis grouped into 5 clusters each having its own aggregated light source.As such, tile 660 n has 5 aggregated light sources used for renderingobjects in that tile. In another example, tile 660 m is shown with moredetail. Tile 660 m may be defined to include a grid of 32×32 pixels, forexample. As previously introduced, a tile may be of various shapes andsizes. As shown, tile 660 m has 30 light sources that affect objectsdrawn within that tile, which is grouped into 5 clusters each having itsown aggregated light source. As such, tile 660 m also has 5 aggregatedlight sources used for rendering objects in that tile.

While specific embodiments have been provided to demonstrate theimplementation of a graphics processor of a video rendering system thatis configured to perform foveated rendering, wherein portions of imagesin a foveal region are rendered with high resolution, and portions inthe peripheral region are rendered with lower resolution (e.g.,rendering using aggregated light sources), these are described by way ofexample and not by way of limitation. Those skilled in the art havingread the present disclosure will realize additional embodiments fallingwithin the spirit and scope of the present disclosure.

It should be noted, that access services, such as providing access togames of the current embodiments, delivered over a wide geographicalarea often use cloud computing. Cloud computing is a style of computingin which dynamically scalable and often virtualized resources areprovided as a service over the Internet. Users do not need to be anexpert in the technology infrastructure in the “cloud” that supportsthem. Cloud computing can be divided into different services, such asInfrastructure as a Service (IaaS), Platform as a Service (PaaS), andSoftware as a Service (SaaS). Cloud computing services often providecommon applications, such as video games, online that are accessed froma web browser, while the software and data are stored on the servers inthe cloud. The term cloud is used as a metaphor for the Internet, basedon how the Internet is depicted in computer network diagrams and is anabstraction for the complex infrastructure it conceals.

A Game Processing Server (GPS) (or simply a “game server”) is used bygame clients to play single and multiplayer video games. Most videogames played over the Internet operate via a connection to the gameserver. Typically, games use a dedicated server application thatcollects data from players and distributes it to other players. This ismore efficient and effective than a peer-to-peer arrangement, but itrequires a separate server to host the server application. In anotherembodiment, the GPS establishes communication between the players andtheir respective game-playing devices to exchange information withoutrelying on the centralized GPS.

Dedicated GPSs are servers which run independently of the client. Suchservers are usually run on dedicated hardware located in data centers,providing more bandwidth and dedicated processing power. Dedicatedservers are the preferred method of hosting game servers for mostPC-based multiplayer games. Massively multiplayer online games run ondedicated servers usually hosted by a software company that owns thegame title, allowing them to control and update content.

Users access the remote services with client devices, which include atleast a CPU, a display and I/O. The client device can be a PC, a mobilephone, a netbook, a PDA, etc. In one embodiment, the network executingon the game server recognizes the type of device used by the client andadjusts the communication method employed. In other cases, clientdevices use a standard communications method, such as html, to accessthe application on the game server over the internet.

Embodiments of the present disclosure may be practiced with variouscomputer system configurations including hand-held devices,microprocessor systems, microprocessor-based or programmable consumerelectronics, minicomputers, mainframe computers and the like. Thedisclosure can also be practiced in distributed computing environmentswhere tasks are performed by remote processing devices that are linkedthrough a wire-based or wireless network.

It should be appreciated that a given video game may be developed for aspecific platform and a specific associated controller device. However,when such a game is made available via a game cloud system as presentedherein, the user may be accessing the video game with a differentcontroller device. For example, a game might have been developed for agame console and its associated controller, whereas the user might beaccessing a cloud-based version of the game from a personal computerutilizing a keyboard and mouse. In such a scenario, the input parameterconfiguration can define a mapping from inputs which can be generated bythe user's available controller device (in this case, a keyboard andmouse) to inputs which are acceptable for the execution of the videogame.

In another example, a user may access the cloud gaming system via atablet computing device, a touchscreen smartphone, or other touchscreendriven device. In this case, the client device and the controller deviceare integrated together in the same device, with inputs being providedby way of detected touchscreen inputs/gestures. For such a device, theinput parameter configuration may define particular touchscreen inputscorresponding to game inputs for the video game. For example, buttons, adirectional pad, or other types of input elements might be displayed oroverlaid during running of the video game to indicate locations on thetouchscreen that the user can touch to generate a game input. Gesturessuch as swipes in particular directions or specific touch motions mayalso be detected as game inputs. In one embodiment, a tutorial can beprovided to the user indicating how to provide input via the touchscreenfor gameplay, e.g. prior to beginning gameplay of the video game, so asto acclimate the user to the operation of the controls on thetouchscreen.

In some embodiments, the client device serves as the connection pointfor a controller device. That is, the controller device communicates viaa wireless or wired connection with the client device to transmit inputsfrom the controller device to the client device. The client device mayin turn process these inputs and then transmit input data to the cloudgame server via a network (e.g., accessed via a local networking devicesuch as a router). However, in other embodiments, the controller canitself be a networked device, with the ability to communicate inputsdirectly via the network to the cloud game server, without beingrequired to communicate such inputs through the client device first. Forexample, the controller might connect to a local networking device (suchas the aforementioned router) to send to and receive data from the cloudgame server. Thus, while the client device may still be required toreceive video output from the cloud-based video game and render it on alocal display, input latency can be reduced by allowing the controllerto send inputs directly over the network to the cloud game server,bypassing the client device.

In one embodiment, a networked controller and client device can beconfigured to send certain types of inputs directly from the controllerto the cloud game server, and other types of inputs via the clientdevice. For example, inputs whose detection does not depend on anyadditional hardware or processing apart from the controller itself canbe sent directly from the controller to the cloud game server via thenetwork, bypassing the client device. Such inputs may include buttoninputs, joystick inputs, embedded motion detection inputs (e.g.,accelerometer, magnetometer, gyroscope), etc. However, inputs thatutilize additional hardware or require processing by the client devicecan be sent by the client device to the cloud game server. These mightinclude captured video or audio from the game environment that may beprocessed by the client device before sending to the cloud game server.Additionally, inputs from motion detection hardware of the controllermight be processed by the client device in conjunction with capturedvideo to detect the position and motion of the controller, which wouldsubsequently be communicated by the client device to the cloud gameserver. It should be appreciated that the controller device inaccordance with various embodiments may also receive data (e.g.,feedback data) from the client device or directly from the cloud gamingserver.

It should be understood that the embodiments described herein may beexecuted on any type of client device. In some embodiments, the clientdevice is a head mounted display (HMD).

FIG. 7, a diagram illustrating components of a head-mounted display 102is shown, in accordance with an embodiment of the disclosure. Thehead-mounted display 102 includes a processor 700 for executing programinstructions. A memory 702 is provided for storage purposes, and mayinclude both volatile and non-volatile memory. A display 704 is includedwhich provides a visual interface that a user may view. A battery 706 isprovided as a power source for the head-mounted display 102. A motiondetection module 708 may include any of various kinds of motionsensitive hardware, such as a magnetometer 710A, an accelerometer 712,and a gyroscope 714.

An accelerometer is a device for measuring acceleration and gravityinduced reaction forces. Single and multiple axis models are availableto detect magnitude and direction of the acceleration in differentdirections. The accelerometer is used to sense inclination, vibration,and shock. In one embodiment, three accelerometers 712 are used toprovide the direction of gravity, which gives an absolute reference fortwo angles (world-space pitch and world-space roll).

A magnetometer measures the strength and direction of the magnetic fieldin the vicinity of the head-mounted display. In one embodiment, threemagnetometers 710A are used within the head-mounted display, ensuring anabsolute reference for the world-space yaw angle. In one embodiment, themagnetometer is designed to span the earth magnetic field, which is ±80microtesla. Magnetometers are affected by metal, and provide a yawmeasurement that is monotonic with actual yaw. The magnetic field may bewarped due to metal in the environment, which causes a warp in the yawmeasurement. If necessary, this warp can be calibrated using informationfrom other sensors such as the gyroscope or the camera. In oneembodiment, accelerometer 712 is used together with magnetometer 710A toobtain the inclination and azimuth of the head-mounted display 102.

A gyroscope is a device for measuring or maintaining orientation, basedon the principles of angular momentum. In one embodiment, threegyroscopes 714 provide information about movement across the respectiveaxis (x, y and z) based on inertial sensing. The gyroscopes help indetecting fast rotations. However, the gyroscopes can drift overtimewithout the existence of an absolute reference. This requires resettingthe gyroscopes periodically, which can be done using other availableinformation, such as positional/orientation determination based onvisual tracking of an object, accelerometer, magnetometer, etc.

A camera 716 is provided for capturing images and image streams of areal environment. More than one camera may be included in thehead-mounted display 102, including a camera that is rear-facing(directed away from a user when the user is viewing the display of thehead-mounted display 102), and a camera that is front-facing (directedtowards the user when the user is viewing the display of thehead-mounted display 102). Additionally, a depth camera 718 may beincluded in the head-mounted display 102 for sensing depth informationof objects in a real environment.

In one embodiment, a camera integrated on a front face of the HMD may beused to provide warnings regarding safety. For example, if the user isapproaching a wall or object, the user may be warned. In one embodiment,the use may be provided with an outline view of physical objects in theroom, to warn the user of their presence. The outline may, for example,be an overlay in the virtual environment. In some embodiments, the HMDuser may be provided with a view to a reference marker, that is overlaidin, for example, the floor. For instance, the marker may provide theuser a reference of where the center of the room is, which in which theuser is playing the game. This may provide, for example, visualinformation to the user of where the user should move to avoid hitting awall or other object in the room. Tactile warnings can also be providedto the user, and/or audio warnings, to provide more safety for when theuser wears and plays games or navigates content with an HMD.

The head-mounted display 102 includes speakers 720 for providing audiooutput. Also, a microphone 722 may be included for capturing audio fromthe real environment, including sounds from the ambient environment,speech made by the user, etc. The head-mounted display 102 includestactile feedback module 724 for providing tactile feedback to the user.In one embodiment, the tactile feedback module 724 is capable of causingmovement and/or vibration of the head-mounted display 102 so as toprovide tactile feedback to the user.

LEDs 726 are provided as visual indicators of statuses of thehead-mounted display 102. For example, an LED may indicate batterylevel, power on, etc. A card reader 728 is provided to enable thehead-mounted display 102 to read and write information to and from amemory card. A USB interface 730 is included as one example of aninterface for enabling connection of peripheral devices, or connectionto other devices, such as other portable devices, computers, etc. Invarious embodiments of the head-mounted display 102, any of variouskinds of interfaces may be included to enable greater connectivity ofthe head-mounted display 102.

A Wi-Fi module 732 is included for enabling connection to the Internetvia wireless networking technologies. Also, the head-mounted display 102includes a Bluetooth module 734 for enabling wireless connection toother devices. A communications link 736 may also be included forconnection to other devices. In one embodiment, the communications link736 utilizes infrared transmission for wireless communication. In otherembodiments, the communications link 736 may utilize any of variouswireless or wired transmission protocols for communication with otherdevices.

Input buttons/sensors 738 are included to provide an input interface forthe user. Any of various kinds of input interfaces may be included, suchas buttons, touchpad, joystick, trackball, etc. An ultra-soniccommunication module 740 may be included in head-mounted display 102 forfacilitating communication with other devices via ultra-sonictechnologies.

Bio-sensors 742 are included to enable detection of physiological datafrom a user. In one embodiment, the bio-sensors 742 include one or moredry electrodes for detecting bio-electric signals of the user throughthe user's skin.

Photo-sensors 744 are included to respond to signals from emitters(e.g., infrared base stations) placed in a 3-dimensional physicalenvironment. The gaming console analyzes the information from thephoto-sensors 744 and emitters to determine position and orientationinformation related to the head-mounted display 102.

The foregoing components of head-mounted display 102 have been describedas merely exemplary components that may be included in head-mounteddisplay 102. In various embodiments of the disclosure, the head-mounteddisplay 102 may or may not include some of the various aforementionedcomponents. Embodiments of the head-mounted display 102 may additionallyinclude other components not presently described, but known in the art,for purposes of facilitating aspects of the present disclosure as hereindescribed.

It will be appreciated by those skilled in the art that in variousembodiments of the disclosure, the aforementioned handheld device may beutilized in conjunction with an interactive application displayed on adisplay to provide various interactive functions. The exemplaryembodiments described herein are provided by way of example only, andnot by way of limitation.

FIG. 8 is a block diagram of a Game System 800, according to variousembodiments of the disclosure. Game System 800 is configured to providea video stream to one or more Clients 810 via a Network 815. Game System800 typically includes a Video Server System 820 and an optional gameserver 825. Video Server System 820 is configured to provide the videostream to the one or more Clients 810 with a minimal quality of service.For example, Video Server System 820 may receive a game command thatchanges the state of or a point of view within a video game, and provideClients 810 with an updated video stream reflecting this change in statewith minimal lag time. The Video Server System 820 may be configured toprovide the video stream in a wide variety of alternative video formats,including formats yet to be defined. Further, the video stream mayinclude video frames configured for presentation to a user at a widevariety of frame rates. Typical frame rates are 30 frames per second, 80frames per second, and 820 frames per second. Although higher or lowerframe rates are included in alternative embodiments of the disclosure.

Clients 810, referred to herein individually as 810A., 810B., etc., mayinclude head mounted displays, terminals, personal computers, gameconsoles, tablet computers, telephones, set top boxes, kiosks, wirelessdevices, digital pads, stand-alone devices, handheld game playingdevices, and/or the like. Typically, Clients 810 are configured toreceive encoded video streams (i.e., compressed), decode the videostreams, and present the resulting video to a user, e.g., a player of agame. The processes of receiving encoded video streams and/or decodingthe video streams typically includes storing individual video frames ina receive buffer of the client. The video streams may be presented tothe user on a display integral to Client 810 or on a separate devicesuch as a monitor or television. Clients 810 are optionally configuredto support more than one game player. For example, a game console may beconfigured to support two, three, four or more simultaneous players.Each of these players may receive a separate video stream, or a singlevideo stream may include regions of a frame generated specifically foreach player, e.g., generated based on each player's point of view.Clients 810 are optionally geographically dispersed. The number ofclients included in Game System 800 may vary widely from one or two tothousands, tens of thousands, or more. As used herein, the term “gameplayer” is used to refer to a person that plays a game and the term“game playing device” is used to refer to a device used to play a game.In some embodiments, the game playing device may refer to a plurality ofcomputing devices that cooperate to deliver a game experience to theuser. For example, a game console and an HMD may cooperate with thevideo server system 820 to deliver a game viewed through the HMD. In oneembodiment, the game console receives the video stream from the videoserver system 820, and the game console forwards the video stream, orupdates to the video stream, to the HMD for rendering.

Clients 810 are configured to receive video streams via Network 815.Network 815 may be any type of communication network including, atelephone network, the Internet, wireless networks, powerline networks,local area networks, wide area networks, private networks, and/or thelike. In typical embodiments, the video streams are communicated viastandard protocols, such as TCP/IP or UDP/IP. Alternatively, the videostreams are communicated via proprietary standards.

A typical example of Clients 810 is a personal computer comprising aprocessor, non-volatile memory, a display, decoding logic, networkcommunication capabilities, and input devices. The decoding logic mayinclude hardware, firmware, and/or software stored on a computerreadable medium. Systems for decoding (and encoding) video streams arewell known in the art and vary depending on the particular encodingscheme used.

Clients 810 may, but are not required to, further include systemsconfigured for modifying received video. For example, a client may beconfigured to perform further rendering, to overlay one video image onanother video image, to crop a video image, and/or the like. Forexample, Clients 810 may be configured to receive various types of videoframes, such as I-frames, P-frames and B-frames, and to process theseframes into images for display to a user. In some embodiments, a memberof Clients 810 is configured to perform further rendering, shading,conversion to 3-D, or like operations on the video stream. A member ofClients 810 is optionally configured to receive more than one audio orvideo stream. Input devices of Clients 810 may include, for example, aone-hand game controller, a two-hand game controller, a gesturerecognition system, a gaze recognition system, a voice recognitionsystem, a keyboard, a joystick, a pointing device, a force feedbackdevice, a motion and/or location sensing device, a mouse, a touchscreen, a neural interface, a camera, input devices yet to be developed,and/or the like.

The video stream (and optionally audio stream) received by Clients 810is generated and provided by Video Server System 820. As is describedfurther elsewhere herein, this video stream includes video frames (andthe audio stream includes audio frames). The video frames are configured(e.g., they include pixel information in an appropriate data structure)to contribute meaningfully to the images displayed to the user. As usedherein, the term “video frames” is used to refer to frames includingpredominantly information that is configured to contribute to, e.g. toeffect, the images shown to the user. Most of the teachings herein withregard to “video frames” can also be applied to “audio frames.”

Clients 810 are typically configured to receive inputs from a user.These inputs may include game commands configured to change the state ofthe video game or otherwise affect gameplay. The game commands can bereceived using input devices and/or may be automatically generated bycomputing instructions executing on Clients 810. The received gamecommands are communicated from Clients 810 via Network 815 to VideoServer System 820 and/or Game Server 825. For example, in someembodiments, the game commands are communicated to Game Server 825 viaVideo Server System 820. In some embodiments, separate copies of thegame commands are communicated from Clients 810 to Game Server 825 andVideo Server System 820. The communication of game commands isoptionally dependent on the identity of the command Game commands areoptionally communicated from Client 810A through a different route orcommunication channel that that used to provide audio or video streamsto Client 810A.

Game Server 825 is optionally operated by a different entity than VideoServer System 820. For example, Game Server 825 may be operated by thepublisher of a multiplayer game. In this example, Video Server System820 is optionally viewed as a client by Game Server 825 and optionallyconfigured to appear from the point of view of Game Server 825 to be aprior art client executing a prior art game engine. Communicationbetween Video Server System 820 and Game Server 825 optionally occursvia Network 815. As such, Game Server 825 can be a prior art multiplayergame server that sends game state information to multiple clients, oneof which is game server system 820. Video Server System 820 may beconfigured to communicate with multiple instances of Game Server 825 atthe same time. For example, Video Server System 820 can be configured toprovide a plurality of different video games to different users. Each ofthese different video games may be supported by a different Game Server825 and/or published by different entities. In some embodiments, severalgeographically distributed instances of Video Server System 820 areconfigured to provide game video to a plurality of different users. Eachof these instances of Video Server System 820 may be in communicationwith the same instance of Game Server 825. Communication between VideoServer System 820 and one or more Game Server 825 optionally occurs viaa dedicated communication channel For example, Video Server System 820may be connected to Game Server 825 via a high bandwidth channel that isdedicated to communication between these two systems.

Video Server System 820 comprises at least a Video Source 830, an I/ODevice 845, a Processor 850, and non-transitory Storage 855. VideoServer System 820 may include one computing device or be distributedamong a plurality of computing devices. These computing devices areoptionally connected via a communications system such as a local areanetwork.

Video Source 830 is configured to provide a video stream, e.g.,streaming video or a series of video frames that form a moving picture.In some embodiments, Video Source 830 includes a video game engine andrendering logic. The video game engine is configured to receive gamecommands from a player and to maintain a copy of the state of the videogame based on the received commands This game state includes theposition of objects in a game environment, as well as typically a pointof view. The game state may also include properties, images, colorsand/or textures of objects.

The game state is typically maintained based on game rules, as well asgame commands such as move, turn, attack, set focus to, interact, use,and/or the like. Part of the game engine is optionally disposed withinGame Server 825. Game Server 825 may maintain a copy of the state of thegame based on game commands received from multiple players usinggeographically disperse clients. In these cases, the game state isprovided by Game Server 825 to Video Source 830, wherein a copy of thegame state is stored and rendering is performed. Game Server 825 mayreceive game commands directly from Clients 810 via Network 815, and/ormay receive game commands via Video Server System 820.

Video Source 830 typically includes rendering logic, e.g., hardware,firmware, and/or software stored on a computer readable medium such asStorage 855. This rendering logic is configured to create video framesof the video stream based on the game state. All or part of therendering logic is optionally disposed within a graphics processing unit(GPU). Rendering logic typically includes processing stages configuredfor determining the three-dimensional spatial relationships betweenobjects and/or for applying appropriate textures, etc., based on thegame state and viewpoint. The rendering logic produces raw video that isthen usually encoded prior to communication to Clients 810. For example,the raw video may be encoded according to an Adobe Flash® standard,.wav, H.264, H.263, On2, VP6, VC-1, WMA, Huffyuv, Lagarith, MPG-x. Xvid.FFmpeg, x264, VP6-8, realvideo, mp3, or the like. The encoding processproduces a video stream that is optionally packaged for delivery to adecoder on a remote device. The video stream is characterized by a framesize and a frame rate. Typical frame sizes include 800×600, 1280×720(e.g., 720p), 1024×768, although any other frame sizes may be used. Theframe rate is the number of video frames per second. A video stream mayinclude different types of video frames. For example, the H.264 standardincludes a “P” frame and a “I” frame. I-frames include information torefresh all macro blocks/pixels on a display device, while P-framesinclude information to refresh a subset thereof. P-frames are typicallysmaller in data size than are I-frames. As used herein the term “framesize” is meant to refer to a number of pixels within a frame. The term“frame data size” is used to refer to a number of bytes required tostore the frame.

In alternative embodiments Video Source 830 includes a video recordingdevice such as a camera. This camera may be used to generate delayed orlive video that can be included in the video stream of a computer game.The resulting video stream optionally includes both rendered images andimages recorded using a still or video camera. Video Source 830 may alsoinclude storage devices configured to store previously recorded video tobe included in a video stream. Video Source 830 may also include motionor positioning sensing devices configured to detect motion or positionof an object, e.g., person, and logic configured to determine a gamestate or produce video-based on the detected motion and/or position.

Video Source 830 is optionally configured to provide overlays configuredto be placed on other video. For example, these overlays may include acommand interface, log in instructions, messages to a game player,images of other game players, video feeds of other game players (e.g.,webcam video). In embodiments of Client 810A including a touch screeninterface or a gaze detection interface, the overlay may include avirtual keyboard, joystick, touch pad, and/or the like. In one exampleof an overlay a player's voice is overlaid on an audio stream. VideoSource 830 optionally further includes one or more audio sources.

In embodiments wherein Video Server System 820 is configured to maintainthe game state based on input from more than one player, each player mayhave a different point of view comprising a position and direction ofview. Video Source 830 is optionally configured to provide a separatevideo stream for each player based on their point of view. Further,Video Source 830 may be configured to provide a different frame size,frame data size, and/or encoding to each of Client 810. Video Source 830is optionally configured to provide 3-D video.

I/O Device 845 is configured for Video Server System 820 to send and/orreceive information such as video, commands, requests for information, agame state, gaze information, device motion, device location, usermotion, client identities, player identities, game commands, securityinformation, audio, and/or the like. I/O Device 845 typically includescommunication hardware such as a network card or modem. I/O Device 845is configured to communicate with Game Server 825, Network 815, and/orClients 810.

Processor 850 is configured to execute logic, e.g. software, includedwithin the various components of Video Server System 820 discussedherein. For example, Processor 850 may be programmed with softwareinstructions in order to perform the functions of Video Source 830, GameServer 825, and/or a Client Qualifier 860. Video Server System 820optionally includes more than one instance of Processor 850. Processor850 may also be programmed with software instructions in order toexecute commands received by Video Server System 820, or to coordinatethe operation of the various elements of Game System 800 discussedherein. Processor 850 may include one or more hardware device. Processor850 is an electronic processor.

Storage 855 includes non-transitory analog and/or digital storagedevices. For example, Storage 855 may include an analog storage deviceconfigured to store video frames. Storage 855 may include a computerreadable digital storage, e.g., a hard drive, an optical drive, or solidstate storage. Storage 855 is configured (e.g., by way of an appropriatedata structure or file system) to store video frames, artificial frames,a video stream including both video frames and artificial frames, audioframe, an audio stream, and/or the like. Storage 855 is optionallydistributed among a plurality of devices. In some embodiments, Storage855 is configured to store the software components of Video Source 830discussed elsewhere herein. These components may be stored in a formatready to be provisioned when needed.

Video Server System 820 optionally further comprises Client Qualifier860. Client Qualifier 860 is configured for remotely determining thecapabilities of a client, such as Clients 810A or 810B. Thesecapabilities can include both the capabilities of Client 810A itself aswell as the capabilities of one or more communication channels betweenClient 810A and Video Server System 820. For example, Client Qualifier860 may be configured to test a communication channel through Network815.

Client Qualifier 860 can determine (e.g., discover) the capabilities ofClient 810A manually or automatically. Manual determination includescommunicating with a user of Client 810A and asking the user to providecapabilities. For example, in some embodiments, Client Qualifier 860 isconfigured to display images, text, and/or the like within a browser ofClient 810A. In one embodiment, Client 810A is an HMD that includes abrowser. In another embodiment, client 810A is a game console having abrowser, which may be displayed on the HMD. The displayed objectsrequest that the user enter information such as operating system,processor, video decoder type, type of network connection, displayresolution, etc., of Client 810A. The information entered by the user iscommunicated back to Client Qualifier 860.

Automatic determination may occur, for example, by execution of an agenton Client 810A and/or by sending test video to Client 810A. The agentmay comprise computing instructions, such as java script, embedded in aweb page or installed as an add-on. The agent is optionally provided byClient Qualifier 860. In various embodiments, the agent can find outprocessing power of Client 810A, decoding and display capabilities ofClient 810A, lag time reliability and bandwidth of communicationchannels between Client 810A and Video Server System 820, a display typeof Client 810A, firewalls present on Client 810A, hardware of Client810A, software executing on Client 810A, registry entries within Client810A, and/or the like.

Client Qualifier 860 includes hardware, firmware, and/or software storedon a computer readable medium. Client Qualifier 860 is optionallydisposed on a computing device separate from one or more other elementsof Video Server System 820. For example, in some embodiments, ClientQualifier 860 is configured to determine the characteristics ofcommunication channels between Clients 810 and more than one instance ofVideo Server System 820. In these embodiments the information discoveredby Client Qualifier can be used to determine which instance of VideoServer System 820 is best suited for delivery of streaming video to oneof Clients 810.

It should be understood that the various embodiments defined herein maybe combined or assembled into specific implementations using the variousfeatures disclosed herein. Thus, the examples provided are just somepossible examples, without limitation to the various implementationsthat are possible by combining the various elements to define many moreimplementations. In some examples, some implementations may includefewer elements, without departing from the spirit of the disclosed orequivalent implementations.

Embodiments of the present disclosure may be practiced with variouscomputer system configurations including hand-held devices,microprocessor systems, microprocessor-based or programmable consumerelectronics, minicomputers, mainframe computers and the like.Embodiments of the present disclosure can also be practiced indistributed computing environments where tasks are performed by remoteprocessing devices that are linked through a wire-based or wirelessnetwork.

With the above embodiments in mind, it should be understood thatembodiments of the present disclosure can employ variouscomputer-implemented operations involving data stored in computersystems. These operations are those requiring physical manipulation ofphysical quantities. Any of the operations described herein that formpart of embodiments of the present disclosure are useful machineoperations. Embodiments of the invention also relate to a device or anapparatus for performing these operations. The apparatus can bespecially constructed for the required purpose, or the apparatus can bea general-purpose computer selectively activated or configured by acomputer program stored in the computer. In particular, variousgeneral-purpose machines can be used with computer programs written inaccordance with the teachings herein, or it may be more convenient toconstruct a more specialized apparatus to perform the requiredoperations.

The disclosure can also be embodied as computer readable code on acomputer readable medium. The computer readable medium is any datastorage device that can store data, which can be thereafter be read by acomputer system. Examples of the computer readable medium include harddrives, network attached storage (NAS), read-only memory, random-accessmemory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes and other optical andnon-optical data storage devices. The computer readable medium caninclude computer readable tangible medium distributed over anetwork-coupled computer system so that the computer readable code isstored and executed in a distributed fashion.

Although the method operations were described in a specific order, itshould be understood that other housekeeping operations may be performedin between operations, or operations may be adjusted so that they occurat slightly different times, or may be distributed in a system whichallows the occurrence of the processing operations at various intervalsassociated with the processing, as long as the processing of the overlayoperations are performed in the desired way.

Although the foregoing disclosure has been described in some detail forpurposes of clarity of understanding, it will be apparent that certainchanges and modifications can be practiced within the scope of theappended claims. Accordingly, the present embodiments are to beconsidered as illustrative and not restrictive, and embodiments of thepresent disclosure is not to be limited to the details given herein, butmay be modified within the scope and equivalents of the appended claims.

What is claimed is:
 1. A method for implementing a graphics pipeline,comprising: determining a plurality of light sources affecting a virtualscene; projecting geometries of objects of an image of the scene onto aplurality of pixels of a display from a first point-of-view;partitioning a plurality of pixels of the display into a plurality oftiles; defining a foveal region of highest resolution for the image asdisplayed, wherein a first subset of pixels is assigned to the fovealregion, and wherein a second subset of pixels is assigned to aperipheral region that is outside of the foveal region; determining afirst set of light sources from the plurality of light sources thataffect one or more objects displayed in a first tile that is in theperipheral region; and clustering at least two light sources from thefirst set into a first aggregated light source affecting the first tilewhen rendering the image in pixels of the first tile.
 2. The method ofclaim 1, further comprising: generating a plurality of clusters of lightsources from the plurality of light sources using a nearest neighborsearch or a k-means clustering technique, wherein each cluster of lightsources is associated with a corresponding aggregated light source,wherein each cluster of light sources comprises at least one lightsource.
 3. The method of claim 2, further comprising: constraining thenumber of aggregated light sources affecting the first tile to a targetnumber; constraining the maximum distance between two light sources in acluster to a maximum distance; and minimizing a plurality of clustererrors for the plurality of clusters of light sources; and for eachcluster of light sources, generating a corresponding aggregated lightsource.
 4. The method of claim 3, wherein the generating a correspondingaggregated light source further comprises: determining an aggregatedlocation of the corresponding aggregated light source by averaginglocations of light sources in the corresponding cluster of lightsources; determining an aggregated intensity of the correspondingaggregated light source by summing the intensities of light sources inthe corresponding cluster of light sources; and determining anaggregated color of the corresponding aggregated light source byperforming a weighted sum of colors for light sources in thecorresponding cluster of light sources.
 5. The method of claim 3,wherein the cluster error corresponding to each cluster in the pluralityof clusters of light sources comprises an iterative sum of the distancesbetween a current light source and each of the other light sources,wherein the current light source comprises a different light source inthe cluster for each iteration.
 6. The method of claim 1, furthercomprising: constraining the number of aggregated light sourcesaffecting the first tile to a target number; determining that the firstset of light sources is less than the target number; and expanding asize of the first tile to meet the target number.
 7. The method of claim1, wherein the display comprises a head mounted display (HMD).
 8. Themethod of claim 1, wherein the foveal region in the image corresponds toa static area that is centered in the display.
 9. The method of claim 1,wherein the foveal region in the image corresponds to where on thedisplay an eye of the user is directed.
 10. A computer systemcomprising: a processor; and memory coupled to the processor and havingstored therein instructions that, if executed by the computer system,cause the computer system to execute a method for implementing agraphics pipeline, comprising: determining a plurality of light sourcesaffecting a virtual scene; projecting geometries of objects of an imageof the scene onto a plurality of pixels of a display from a firstpoint-of-view; partitioning a plurality of pixels of the display into aplurality of tiles; defining a foveal region of highest resolution forthe image as displayed, wherein a first subset of pixels is assigned tothe foveal region, and wherein a second subset of pixels is assigned toa peripheral region that is outside of the foveal region; determining afirst set of light sources from the plurality of light sources thataffect one or more objects displayed in a first tile that is in theperipheral region; and clustering at least two light sources from thefirst set into a first aggregated light source affecting the first tilewhen rendering the image in pixels of the first tile.
 11. The computersystem of claim 10, wherein the method further comprises: generating aplurality of clusters of light sources from the plurality of lightsources using a nearest neighbor search or a k-means clusteringtechnique, wherein each cluster of light sources is associated with acorresponding aggregated light source, wherein each cluster of lightsources comprises at least one light source.
 12. The computer system ofclaim 11, wherein the method further comprises: constraining the numberof aggregated light sources affecting the first tile to a target number;constraining the maximum distance between two light sources in a clusterto a maximum distance; and minimizing a plurality of cluster errors forthe plurality of clusters of light sources; and for each cluster oflight sources, generating a corresponding aggregated light source. 13.The computer system of claim 12, wherein the generating a correspondingaggregated light source further comprises: determining an aggregatedlocation of the corresponding aggregated light source by averaginglocations of light sources in the corresponding cluster of lightsources; determining an aggregated intensity of the correspondingaggregated light source by summing the intensities of light sources inthe corresponding cluster of light sources; and determining anaggregated color of the corresponding aggregated light source byperforming a weighted sum of colors for light sources in thecorresponding cluster of light sources.
 14. The computer system of claim12, wherein the cluster error corresponding to each cluster in theplurality of clusters of light sources comprises an iterative sum of thedistances between a current light source and each of the other lightsources, wherein the current light source comprises a different lightsource in the cluster for each iteration.
 15. The computer system ofclaim 10, wherein the method further comprises: constraining the numberof aggregated light sources affecting the first tile to a target number;determining that the first set of light sources is less than the targetnumber; and expanding a size of the first tile to meet the targetnumber.
 16. The computer system of claim 10, wherein the displaycomprises a head mounted display (HMD).
 17. A non-transitorycomputer-readable medium storing a computer program for implementing agraphics pipeline, the computer-readable medium comprising: programinstructions for determining a plurality of light sources affecting avirtual scene; program instructions for projecting geometries of objectsof an image of the scene onto a plurality of pixels of a display from afirst point-of-view; program instructions for partitioning a pluralityof pixels of the display into a plurality of tiles; program instructionsfor defining a foveal region of highest resolution for the image asdisplayed, wherein a first subset of pixels is assigned to the fovealregion, and wherein a second subset of pixels is assigned to aperipheral region that is outside of the foveal region; programinstructions for determining a first set of light sources from theplurality of light sources that affect one or more objects displayed ina first tile that is in the peripheral region; and program instructionsfor clustering at least two light sources from the first set into afirst aggregated light source affecting the first tile when renderingthe image in pixels of the first tile.
 18. The non-transitorycomputer-readable medium of claim 17, further comprising: programinstructions for generating a plurality of clusters of light sourcesfrom the plurality of light sources using a nearest neighbor search or ak-means clustering technique, wherein each cluster of light sources isassociated with a corresponding aggregated light source, wherein eachcluster of light sources comprises at least one light source.
 19. Thenon-transitory computer-readable medium of claim 28, further comprising:program instructions for constraining the number of aggregated lightsources affecting the first tile to a target number; programinstructions for constraining the maximum distance between two lightsources in a cluster to a maximum distance; and program instructions forminimizing a plurality of cluster errors for the plurality of clustersof light sources; and for each cluster of light sources, programinstructions for generating a corresponding aggregated light source. 20.The non-transitory computer-readable medium of claim 19, wherein theprogram instructions for generating a corresponding aggregated lightsource further comprises: program instructions for determining anaggregated location of the corresponding aggregated light source byaveraging locations of light sources in the corresponding cluster oflight sources; program instructions for determining an aggregatedintensity of the corresponding aggregated light source by summing theintensities of light sources in the corresponding cluster of lightsources; and program instructions for determining an aggregated color ofthe corresponding aggregated light source by performing a weighted sumof colors for light sources in the corresponding cluster of lightsources.
 21. The non-transitory computer-readable medium of claim 19,wherein the cluster error corresponding to each cluster in the pluralityof clusters of light sources comprises an iterative sum of the distancesbetween a current light source and each of the other light sources,wherein the current light source comprises a different light source inthe cluster for each iteration.